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Abstract 

We  present  a  convergence  theory  for  pattern  search  methods  for  solving  bound  con¬ 
strained  nonlinear  programs.  The  analysis  relies  on  the  abstract  structure  of  pattern 
search  methods  and  an  understanding  of  how  the  pattern  interacts  with  the  bound  con¬ 
straints.  This  analysis  makes  it  possible  to  develop  pattern  search  methods  for  bound 
constrained  problems  while  only  slightly  restricting  the  flexibility  present  in  pattern 
search  methods  for  unconstrained  problems.  We  prove  global  convergence  despite  the 
fact  that  pattern  search  methods  do  not  have  explicit  information  concerning  the  gra¬ 
dient  and  its  projection  onto  the  feasible  region  and  consequently  are  unable  to  enforce 
explicitly  a  notion  of  sufficient  feasible  decrease. 
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1.  Introduction.  This  paper  extends  the  class  of  pattern  search  methods  for  un¬ 
constrained  minimization,  considered  in  [14],  to  bound  constrained  problems: 

minimize  f{x) 
subject  to  i  <  X  <  u, 

where  /  :  R”  — >  R,  £,x,u  €  R”,  and  £  <  u.  We  allow  the  possibility  that  some  of  the 
variables  are  unbounded  either  above  or  below  by  permitting  £j,Uj  =  ±oo,  y  =  1, . . . ,  n. 

Our  convergence  analysis  is  guided  by  that  for  pattern  search  methods  for  un¬ 
constrained  problems  [14].  We  can  guarantee  that  if  the  objective  /  is  continuously 
differentiable,  then  a  subsequence  of  the  iterates  produced  by  a  pattern  search  method 
for  problems  with  bound  constraints  converges  to  a  stationary  point  of  problem  (1). 
By  a  stationary  point  of  problem  (1)  we  mean  a  feasible  point  x  that  satisfies  the  first- 
order  necessary  condition  for  optimality:  for  all  feasible  z  e  R”,  {Vf{x) ,  z  -  x)  >  0. 
Equivalently,  a;  is  a  Karush-Kuhn-Tucker  point  for  problem  (1).  As  in  the  case  of 
unconstrained  minimization,  pattern  search  methods  for  bound  constrained  problems 
accomplish  this  without  an  explicit  representation  of  the  gradient  or  the  directional 
derivative.  In  particular,  we  prove  global  convergence  in  the  bound  constrained  case 
even  though  pattern  search  methods  do  not  have  explicit  information  concerning  the 
gradient  and  its  projection  onto  the  feasible  region  and  consequently  do  not  explicitly 
enforce  a  notion  of  sufficient  feasible  decrease. 

The  general  specification  of  pattern  search  methods  for  bound  constrained  mini¬ 
mization  gives  one  broad  latitude  in  designing  such  algorithms.  Moreover,  as  we  shall 
discuss,  classical  pattern  search  methods  for  unconstrained  minimization — such  as  co¬ 
ordinate  search  with  fixed  step  sizes  and  the  original  pattern  search  of  Hooke  and  Jeeves 
[9] — can  be  generalized  without  modification  to  the  bound  constrained  case.  We  also  will 
show  that  not  all  pattern  search  methods  for  unconstrained  minimization  immediately 
generalize  to  bound  constrained  problems:  in  §5.2  we  will  present  a  counterexample 
that  defeats  G.E.P.  Box’s  method  of  evolutionary  operation  using  two-level  factorial 
designs  [1],  [-3,  12]  and  show  how  the  convergence  theory  guides  us  to  a  remedy  that 
uses  composite  designs  [2],  instead  of  the  simpler  factorial  or  fractional  factorial  designs. 
The  multidirectional  search  algorithm  of  Dennis  and  Torczon  [7,  13]  also  requires  us  to 
augment  the  pattern  used  for  the  algorithm;  again  we  find  a  straightforward  exten¬ 
sion,  but  one  that  reveals  much  about  the  interesting  behavior  of  the  simplices  which 
characterize  that  method. 

Notation.  We  denote  by  R,  Q,  Z,  and  N  the  sets  of  real,  rational,  integer,  and 
natural  numbers,  respectively. 

Unless  otherwise  noted,  norms  are  assumed  to  be  the  Euclidean  norm.  The  feasible 
region  for  problem  (1)  we  denote  by  fi: 

fl  =  {  a;  €  R”  |  £  <  x  <u} . 


1 


The  projection  onto  0,  we  denote  by  P.  If  for  scalar  t  we  define 

ij  if  t  <  ij 

Pj{t)  =  <  t  ii  ij  <  t  <  Uj 

Uj  if  t  >  Uj, 

then  the  projection  of  a:  =  (a:i,  •  •  • ,  is  given  by 

n 

p{x)  = 

j  =  l 

where  {ej},  j  =  1,  -  ■  ■  ,n  are  the  standard  basis  vectors.  On  those  few  occasions  where 
we  must  denote  components  of  subscripted  vectors,  we  use  the  following  notation:  qk,j 
denotes  the  component  of  the  vector  qk- 

We  will  denote  by  g{x)  the  gradient  V/(a;)  of  the  objective.  Finally,  let 

Lu{y)  =  {  X  G  0  I  f{x)  <  f{y)  }. 

2.  Pattern  Search  Methods.  We  begin  by  defining  the  general  pattern  search 
method  for  the  bound  constrained  problem  (1);  it  differs  from  that  for  unconstrained 
problems  [14]  in  only  a  few  particulars,  which  we  summarize  in  §2.5. 

2,1.  The  Pattern.  As  with  pattern  search  methods  for  unconstrained  problems, 
to  define  a  pattern  we  need  two  components:  a  basis  matrix  and  a  generating  matrix. 
The  basis  matrix  is  a  nonsingular  matrix  B  G 

The  generating  matrix  is  a  matrix  Ck  €  where  p  >  2n.  We  partition  the 

generating  matrix  into  components 

(2)  Ck  =  [  Mk  -Mk  Lk  ]  =  [  Tk  Lk  ]. 

We  require  that  G  M  C  Z”^”,  where  M  is  a  finite  set  of  nonsingular  matrices,  and 

that  Lk  €  and  that  it  contains  at  least  one  column,  a  column  of  zeroes. 

A  pattern  Pk  is  then  defined  by  the  columns  of  the  matrix  Pk  =  BCk-  For  conve¬ 
nience,  we  use  the  partition  of  the  generating  matrix  Ck  given  in  (2)  to  partition  Pk  as 
follows: 

Pk  =  BCk  =  [  BMk  -BMk  BLk  ]  ^  [  BTk  BLk  ]. 

We  also  require  the  matrix  BMk  to  be  diagonal: 

(3)  BMk  =  diag(o?fc),  ?  =  1, . . . ,  n. 

This  condition,  absent  in  the  case  of  unconstrained  minimization,  is  needed  in  order  to 
ensure  that  we  can  find  feasible  points  in  the  pattern  that  will  also  produce  decrease  in 
the  objective.  As  we  shall  see,  this  condition  is  not  especially  restrictive  and  is  satis¬ 
fied  by  all  of  the  commonly  encountered  pattern  search  algorithms  or  straightforward 
variants  of  them. 

Given  Ajt  G  R,  >  0,  we  define  a  trial  step  si  to  be  any  vector  of  the  form 
si  =  AkBPk,  where  4  denotes  a  column  of  Ck  =  [4 •••4].  We  call  a  trial  step  si 

feasible  if  (xk+si)  G  fi.  At  iteration  k,  a  trial  point  is  any  point  of  the  form  xj,  =  Xk+sl, 

where  x;,  is  the  current  iterate. 


2 


2.2.  The  Bound  Constrained  Exploratory  Moves.  Pattern  search  methods 
proceed  by  conducting  a  series  of  exploratory  moves  about  the  current  iterate  Xk  to 
choose  a  new  iterate  x^+i  =  Xk  +  Sk,  for  some  feasible  step  Sk  determined  during  the 
course  of  the  exploratory  moves.  The  following  Hypotheses  on  Bound  Constrained 
Exploratory  Moves  allows  a  broad  choice  of  exploratory  moves  while  ensuring  the  prop¬ 
erties  required  to  prove  convergence.  By  abuse  of  notation,  if  A  is  a  matrix,  y  ^  A 
means  that  the  vector  j/  is  a  column  of  A. 

Hypotheses  on  Bound  Constrained  Exploratory  Moves. 

1.  G  AkPk  =  A>.kBCk  =  Ajt  [BTk  BLk\. 

2.  {xk  "t*  Sk)  G  fl. 

3.  If  min  {  f{xk  +y)  \  ye  AkBTk,  xkAye9.}<  /(xk), 
then  f(xk+ Sk)  <  f{xk). 

2.3.  The  Generalized  Pattern  Search  Method.  Algorithm  1  states  the  gen¬ 
eralized  pattern  search  method  for  minimization  with  bound  constraints.  To  define  a 
particular  pattern  search  method,  we  must  specify  the  basis  matrix  B,  the  generating 
matrix  Cfc,  the  bound  constrained  exploratory  moves  to  be  used  to  produce  a  feasible 
step  Skt  and  the  algorithms  for  updating  Ck  and  Afc. 

Algorithm  1.  The  Generalized  Pattern  Search  Method  for  Bound  Constrained  Prob¬ 
lems. 

Let  xo  €  0  and  Ao  >  0  be  given. 

For  A;  =  0, 1,  •  •  • , 

a)  Compute  f{xk)- 

b)  Determine  a  step  Sk  using  a  bound  constrained  exploratory  moves  algorithm. 

c)  Compute  pk  =  f{xk)  -  f{xk  +  Sk). 

d)  If  pfc  >  0  then  Xk+i  =  Xk  +  Sk-  Otherwise  Xk-\-\  —  Xk- 

e)  Update  Ck  and  A^. 

2.4.  The  Updates.  Algorithm  2  specifies  the  requirements  for  updating  Ajt.  The 
aim  of  the  update  of  Ak  is  to  force  pk  >  0.  An  iteration  with  p*  >  0  is  successful; 
otherwise,  the  iteration  is  unsuccessful.  Note  that  to  accept  a  step  we  only  require 
simple,  as  opposed  to  sufficient,  decrease. 

Algorithm  2.  Updating  A*. 

Let  T  G  Q,  T  >  1,  and  •  •  • ,  wl}  C  Z,  wq  <  0,  and  Wi  >  0,  i  =  1,  -  ■  ■  ,L.  Let 

^  =  r“'°,  and  Afc  G  A  =  {r“'S •  •  • , r^i}. 

a)  If  Pfc  <  0  then  A^+i  =  6Ak. 

b)  If  pfc  >  0  then  A^+i  =  XkAk- 

The  conditions  on  0  and  A  ensure  that  0  <  ^  <  1  and  \i  >  1  for  all  A,-  G  A.  Thus, 
if  an  iteration  is  successful  it  may  be  possible  to  increase  the  step  length  parameter  A^, 
but  Ak  is  not  allowed  to  decrease. 
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2.5.  Differences  between  Pattern  Search  Methods  for  Unconstrained  and 
Bound  Constrained  Minimization.  There  are  only  two  additional  restrictions  re¬ 
quired  of  pattern  search  methods  to  ensure  convergence  for  the  bound  constrained  case. 

First,  we  note  that  pattern  search  methods  are  feasible  point  methods;  the  search 
begins  with  a  point  that  satisfies  the  bounds  and  maintains  feasibility  throughout  the 
search.  This  can  be  seen  in  Algorithm  1,  where  we  require  xq  E  0.  This  requirement 
also  appears  in  the  Hypotheses  on  Bound  Constrained  Exploratory  Moves;  if  simple 
decrease  on  the  function  value  at  the  current  iterate  can  be  found  among  any  of  the 
feasible  trial  steps  contained  in  the  columns  of  AkBTk,  then  the  exploratory  moves 
must  produce  a  feasible  step  Sk  that  also  gives  simple  decrease  on  the  function  value  at 
the  current  iterate. 

The  second,  and  more  interesting,  restriction  is  that  the  core  pattern  BMk  must 
be  defined  by  a  diagonal  matrix.  Because  the  columns  of  the  pattern  matrix  determine 
the  directions  of  the  steps  that  may  be  considered,  we  need  to  ensure  that  if  we  are 
not  at  a  constrained  stationary  point,  we  have  at  least  one  feasible  direction  of  descent. 
Practically,  we  must  ensure  that  we  have  directions  that  allow  us  to  move  parallel  to 
the  constraints.  Requiring  BMk  to  be  a  diagonal  matrix  is  sufficient.  And  as  we  will 
show  in  §5.2,  this  requirement  is  unavoidable. 

We  note  an  equivalence  between  pattern  search  methods  for  bound  constrained 
problems  and  an  exact  penalization  approach  to  problem  (1).  Applying  a  pattern 
search  method  for  problem  (1)  produces  exactly  the  same  iterates  as  applying  such  an 
algorithm  to  the  unconstrained  problem 

minimize  F{x), 


where 


(4) 


F(^) 


f{x)  if  a;  G 
oo  otherwise. 


In  fact,  this  is  one  classical  approach  used  with  direct  search  methods  to  ensure  that 
the  iterates  produced  remain  feasible  (see,  for  instance,  [10,  11]).  In  the  case  of  pattern 
search  methods  this  formulation  is  not  simply  a  conceptual  approach;  pattern  search 
methods  are  directly  applicable  to  this  exact  penalty  function  since  they  do  not  rely 
on  derivatives.  However,  as  we  will  demonstrate  in  §5.2,  this  exact  penalization  ap¬ 
proach  cannot  be  applied  with  an  arbitrary  pattern  search  method  for  unconstrained 
minimization;  we  must  require  that  BMk  be  diagonal. 

2.6.  Results  from  the  Unconstrained  Theory.  We  recall  the  following  results 
from  [14],  to  which  we  refer  the  reader  for  the  proofs.  The  first  result  indicates  one 
sense  in  which  regulates  step  length. 

Lemma  2.1  (Lemma  3.1  from  [14]).  There  exists  a  constant  C*  >  0,  independent 
of  k,  such  that  for  any  trial  step  s\  produced  by  a  generalized  pattern  search  method 
(Algorithm  1)  we  have  ||  s).  ||  >  C*Ak. 

The  next  result  is  key  to  the  convergence  of  pattern  search  methods.  It  states  that 
the  iterates  produced  by  a  pattern  search  method  have  a  rigid  algebraic  structure. 
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Theorem  2.2  (Theorem  3.2  from  [14]).  Any  iterate  xn  produced  by  a  gener¬ 
alized  pattern  search  method  (Algorithm  1 )  can  he  expressed  in  the  following  form: 

(5)  XN  =  Xo+{!3^^^a-^^^)AoB'£zk, 

k=0 

where 

•  xo  is  the  initial  guess, 

•  /3/a  =  T,  with  a,  €  N  and  relatively  prime,  and  r  is  as  defined  in  the 
algorithm  for  updating  (Algorithm  2), 

•  r IB  fUB  depend  on  N , 

•  Ao  is  the  initial  choice  for  the  step  length  control  parameter, 

•  B  is  the  basis  matrix,  and 

•  Zk  e  Z”,  k  =  0,  -  ■  ■  ,N  -  1. 

The  last  result  we  recollect  says,  in  conjunction  with  Lemma  2.1,  that  if  we  bound 
the  size  of  the  elements  of  the  generating  matrix  (which  is  a  reasonable  thing  to  do), 
then  Ak  completely  regulates  the  size  of  the  steps  a  pattern  search  method  takes. 

Lemma  2.3  (Lemma  3.6  from  [14]).  If  there  exists  a  constant  C  >0  such  that 
for  all  k,C  >  ||c^||,  for  all  i  =  1,  -  •  ■  ,p,  then  there  exists  a  constant  V’*  >  0,  independent 
of  k,  such  that  for  any  trial  step  s\  produced  by  a  generalized  pattern  search  method 
(Algorithm  1)  we  have  Ak  >  V’*||'5];||- 

3.  Convergence  Theory.  We  now  present  the  first-order  constrained  stationary 
point  convergence  theory  for  pattern  search  methods  for  bound  constrained  problems. 
We  begin  by  defining,  for  feasible  x,  the  quantity 

q{x)  =  P{x  -g{x))  -  X. 

In  the  bound  constrained  theory  the  quantity  q{x)  plays  the  role  of  g{x)  in  the  un¬ 
constrained  theory,  giving  us  a  continuous  measure  of  how  close  we  are  to  constrained 
stationarity,  as  in  the  theory  for  methods  based  explicitly  on  derivatives  (e.g.,  [6],  where 
we  got  the  idea).  The  following  proposition  summarizes  properties  of  q  that  we  will 
shortly  need,  particularly  the  fact  that  a:  is  a  constrained  stationary  point  for  (1)  if  and 
only  if  q{x)  =  0.  While  stated  for  the  particular  domain  fl,  the  proposition  holds  for 
any  closed  convex  domain. 

Proposition  3.1.  Let  x  €9,.  Then 

(6)  II  9(^)  II  <  II  9{^)  II , 

(7) 

and 

(8)  X  is  a  stationary  point  for  problem  (1)  if  and  only  if  q{x)  =  0. 


II  <  II  P{9{x)) 
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Proof.  We  need  the  following  properties  of  the  projection; 

(a)  If  z  G  0  then  {P{x)  —  x,z  —  P{x))  >  0  for  all  x  G  R”. 

(b)  For  all  x,y  G  R”,  ||  P{y)  -  P{x)  \\  <  \\  y  -  x  |1. 

(c)  If  T{x)  is  the  tangent  cone  of  0,  at  x,  then 

min{  |  v  G  T{x),  ||  u  ||  <  1  }  =  -  ||  Pig{x))  || . 

The  proofs  of  (a)  and  (b)  may  be  found  in  [8],  and  the  proof  of  (c)  may  be  found  in 
[5]  (there  is  a  difference  in  sign  between  Pigix))  here  and  the  quantity  Va/(x)  as  it  is 
defined  in  the  latter  reference). 

We  first  prove  (6).  If  a;  G  0,  then  P{x)  =  x  and  so  from  (b), 

II  q{x)  II  =  II  P{x  -  g{x))  -  P{x)  ||  <  ||  (a;  -  g{x))  -  a;  ||  =  ||  g{x)  || , 

which  is  (6). 

Next  we  establish  (7).  Since  P{x-g{x))-x  G  T{x),  we  obtain  from  (c)  the  estimate 
{g{x)  ,  P{x  -  g{x))  -  a;)  >  -  II  P{gix))  ||  ||  P{x  -  ^'(a;))  -  x  || 


or 

(9)  (^(a;)  ,  ?(a:))  >  -  ||  P(^(a;))  ||  ||  q{x)  || . 

Meanwhile,  from  (a)  we  obtain 

{q{x)  +  g{x)  ,  -q{x))  =  {P{x  -  g{x))  -  {x  -  g{x))  ,  x  -  P{x  -  g{x)))  >  0, 
whence 

(10)  (^(x)  ,  ^(a;))  <  -  II  9(x)  ||\ 

Combining  (9)  and  (10)  we  arrive  at 

II  9(0;)  II  <  II  P(5(a:))  II , 


which  is  (7). 

Now  consider  (8).  First  suppose  that  a:  is  a  stationary  point  for  problem  (1).  Then 
P{g{x))  =  0,  so  from  (7)  it  immediately  follows  that  q{x)  =  0.  Conversely,  suppose 
that  9(a;)  =  0.  Then  P(a;  —  ^(a;))  =  x,  so  from  (a)  we  have,  for  all  2;  G  fi, 

(P(a;  -  g{x))  -  {x  -  g{x))  ,  2;  -  P(ar  -  ^(a;)))  >  0, 


or 


{g{x)  ,  2  -  x)  >  0. 

But  this  is  nothing  other  than  the  first-order  necessary  condition  for  optimality  for 
problem  (1)  at  x.  □ 
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We  can  now  state  the  first  convergence  result  for  the  general  pattern  search  method 
for  bound  constrained  minimization. 

Theorem  3.2.  Let  Lq{xo)  be  compact  and  suppose  f  is  continuously  differentiable 
on  Lq{xo).  Let  be  the  sequence  of  iterates  produced  by  a  generalized  pattern  search 
method  for  bound  constrained  minimization  (Algorithm  1).  Then 

lim  inf  |(  q{xk)  ||  =  0 . 

A:— >-+00 


The  proof  of  this  theorem  is  given  in  §4.1,  after  we  have  established  the  necessary 
intermediate  results. 

We  can  strengthen  the  result  given  in  Theorem  3.2  in  the  same  way  that  we  do  in 
the  unconstrained  case  [14].  First,  we  require  the  columns  of  the  generating  matrix  Ck 
to  remain  bounded  in  norm,  i.e.,  that  there  exists  a  constant  C  >  0  such  that  for  all 
k,  C  >  ||c®^||,  for  all  z  =  1,  •  •  •  ,p.  Second,  we  replace  the  original  Hypotheses  on  Bound 
Constrained  Exploratory  Moves  with  a  stronger  version,  given  below.  Third,  we  require 
that  limjt^+co  =  0.  All  the  algorithms  described  in  §5,  except  multidirectional 
search,  satisfy  this  third  condition  because  of  the  customary  choice  of  A  =  {1}  =  {t°}. 
However,  it  is  not  necessary  to  force  the  steps  to  be  non-increasing. 

Strong  Hypotheses  on  Bound  Constrained  Exploratory  Moves. 

1.  s/t  G  As.kPk  =  AkBCk  =  Ajt  [BVk  BLk]. 

2.  {xk  +  Sk)  €  0. 

3.  If  min{  f{xk  +  y)  \  y  E  Afc^F*,  Xk  +  y  eLt}  <  f{xk), 

then  f{xk  +  Sk)  <  min  {  f{xk  Py)  j  y  E  AkBTk,  Xk  +  y  E9.]. 

Theorem  3.3.  Let  Lffxo)  be  compact  and  suppose  f  is  continuously  differentiable 
on  Lq(xo).  In  addition,  assume  that  the  columns  of  the  generating  matrices  are  uni¬ 
formly  bounded  in  norm,  that  limjt_,+oo  A^  =  0,  and  that  the  generalized  pattern  search 
method  for  bound  constrained  minimization  (Algorithm  1 )  enforces  the  Strong  Hypothe¬ 
ses  on  Bound  Constrained  Exploratory  Moves.  Then  for  the  sequence  of  iterates  {a;^} 
produced  by  the  generalized  pattern  search  method  for  hound  constrained  minimization, 

lim  |k(rr*)||  =  0. 

/c^+oo 


The  proof  will  be  found  in  §4.2. 

4.  Proof  of  Theorems  3.2  and  3.3.  Given  an  iterate  Xk,  let  gk  =  g{xk)  and 
qk  =  q{xk).  Let  w  denote  the  following  modulus  of  continuity  of  g\  given  a;  €  R”  and 
£  >  0, 

uj[x,£)  =  sup  {  6  >  0  I  II  g{y)  —  g{x)  ||  <  £  for  all  y  such  that  ||  j/  —  a:  ||  <  ^  }  . 

We  begin  with  an  elementary  proposition  concerning  descent  directions. 


Proposition  4.1.  Let  d  €  R"  and  suppose  that  f  restricted  to  the  closed  line 
segment  connecting  x  and  x+d  is  continuously  differentiable.  Assume,  too,  that  g{x)  /  0 
and  g{x)^d  <  — £||  c?  ||-  Then,  if\\  c?  ||  <  c<;(a;,  |), 

f{x  +  d)-f{x)  <-^\\d\\. 

Proof.  From  the  mean-value  theorem,  we  have,  for  some  y  on  the  line  segment 
between  x  and  x  +  d, 

f{x  +  d)-f{x)  =  g{xfd+{g{y)-gix)f  d 

<  -s\\d\\  +  \\9{y)-9{^)\\\\d\\- 


If  II  d  II  <  oj{x,  |),  then  ||  g{y)  -  g{x)  ||  <  |  and  the  result  follows.  □ 

In  the  proof  of  the  next  result  the  bound  constrained  and  the  unconstrained  cases 
most  differ.  The  proof  of  Proposition  4.2  implicitly  relies  on  the  fact  that  in  the  bound 
constrained  case,  the  directions  in  the  pattern  defined  by  the  columns  of  BMk  are 
coordinate  directions  and  thus  are  oriented  normal  and  tangent  to  the  faces  of  the 
feasible  region.  That  this  is  not  merely  convenient  is  made  clear  by  the  counterexample 
in  §5.2. 

Proposition  4.2.  Suppose  that  f  is  continuously  differentiable  on  Lq{xo)  and 
q{xk)  ^  0.  Then  there  exists  avk>Q  such  that  if  Ak  <  Vk,  then  there  is  a  trial  .step  si 
defined  by  a  column  of  AkBTk  for  which  {xk  -|-  5).)  £  17  and 


9k  9k 
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where  Cn  =  n  2 . 

Proof.  We  restrict  our  attention  to  the  steps  defined  by  the  columns  of  AkBTk;  by 
hypothesis,  AkBTk  =  AkB[Mk  -Mk]  =  Aife[diag(4)  -  diag(4)]  (see  (3)).  Choose  an 
index  m  for  which 


(11) 


I  9k, m  I  —  II  Iloo  —  /r  II  II  ’ 

\/n 


where  qk,Tn  is  the  component  of  qk-  Note  that  it  is  also  the  case  that 
(12)  1  9k, m  I  ^  I  9k,m  | 

and  sign(5rfc,TO)  =  sign(?fc,m)- 

Let  si  =  —  sign{gk,m)Ak  |  df  \  6^,  this  vector  will  be  among  the  columns  of  AkBTk- 
Since  Xk  +  qk  =  P{xk  —  9k)  is  feasible,  we  have  £  <  Xk  +  qk  <  u  and  thus 


,7n  T  9k,m  ^  ^m- 


It  follows  that  if  Aj,  1  I  <  1  qk,m  |,  then  the  trial  point  xl  =  Xk  +  si  will  be  feasible. 
Moreover,  from  (11)  and  (12), 


9k  4  = 


=  -  Slgn{9k,m)£^k  I  d™  1  9k,m  =  -  4  I  9k,m  |  < 


Sk 


9k 


Defining  Uk  =  ||  qk  ilo^  /  I  then  does  the  trick.  □ 

Proposition  4.3.  Suppose  that  Lq(xo)  is  compact  and  that  f  is  continuously 
differentiable  on  Lq(xo).  Then  given  any  tj  >  0,  there  exists  8  >  0.  independent  of  k, 
such  that  if  Ak  <  6  and  ||  q{xk)  ||  >  Tj,  the  pattern  search  method  for  bound  constrained 
minimization  will  find  an  acceptable  step  Sk;  i.e.,  f{xk-\-Sk)  <  f{xk)  and  (xk  +  Sk)  6  D. 

If,  in  addition,  the  columns  of  the  generating  matrix  remain  bounded  in  norm  and 
we  enforce  the  Strong  Hypotheses  on  Bound  Constrained  Exploratory  Moves,  then,  given 
any  77  >  0,  there  exist  ^  >  0  and  <7  >  0,  independent  of  k,  such  that  if  Ak  <  8  and 
II  q{xk)  II  >  T),  then 


f{xk+i)  <  f{xk)  -  (7  II  q{xk)  II  II  Sk  II . 

Proof  Since  g{x)  is  uniformly  continuous  on  Lq{xo),  there  exists  w.  >  0  such  that 

lx)  (xk,n~^y>u!^ 

for  all  k  for  which  ||  qk  ||  >  q. 

Next,  choose  d*  >  0  such  that  <  d*  for  all  i  and  k.  This  we  can  do  because  the 
set  {d\}  is  finite  (see  (3)  and  the  conditions  on  Mk  given  in  §2.1).  Let 


then 


i  7  ’ 
712(1* 


l/^ 


V  <  II  gfc  II  II  <ik  Hoc 

n^d*  n^d*  ~  d* 


for  all  k  for  which  ||  qk  ||  >  77,  where  is  as  in  Proposition  4.2. 
Finally,  let 


8  =  min(z/,,a;*/d*) . 

Now  suppose  II  q{xk)  ||  >  77  and  <  <5.  Since  Ak  <  Vk,  Proposition  4.2  assures  us  of 
the  existence  of  a  step  s*  defined  by  a  column  of  AkBTk  such  that  {xk  +  s}.)  e  H  and 


9k  ^k  —  ^n  \  qk 


At  the  same  time,  we  also  have 


<  Akd*  <lx>^  <lx>  {xk,  Cn  II  qk  II) . 


So  by  Proposition  4.1, 


f{xk^  si)-  f{xk)  <-^\\qk 


Thus,  when  Ajt  <  8,  f{xl)  =  f[xk  +  $1)  <  f{xk)  for  at  least  one  feasible  $1  €  AkBTk- 
The  Hypotheses  on  Bound  Constrained  Exploratory  Moves  guarantee  that  if 


min {  f{xk  +  y)  |  ?/  €  AkBTk,  Xk-{-yen}<  f{xk), 
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then  f{xk  +  Sk)  <  /(xfc)  and  (xit  +  5fc)  €  This  proves  the  first  part  of  the  Proposition. 

If,  in  addition,  we  enforce  the  Strong  Hypotheses  on  Bound  Constrained  Ex¬ 
ploratory  Moves,  then  we  actually  have 

/(xfc+i)  - /(xfc)  <  “Y  II  II  4  . 

Lemma  2.1  then  ensures  that 

f{xk+i)<f{xk)-jC*^k\\q{xk)\\- 

Applying  Lemma  2.3,  we  arrive  at 

fixk+i)  <  f{xk)-<x\\  q{xk)  II II  Sk  II, 


where  a  =  D 

Corollary  4.4.  Suppose  that  Lq{xo)  is  compact  and  that  f  is  continuously  dif- 
ferentiable  on  Lni,Xo)-  Suppose,  too,  that  limmik-*+oo  ||  ||  7^  0.  Then  there  exists 

a  constant  A*  >  0  such  that  for  all  k,  Aa,  >  A*. 

Proof  By  hypothesis,  there  exists  N  and  77  >  0  such  that  for  all  k  >  N,  ||  q{xk)  ||  > 
Tf.  By  Proposition  4.3,  we  can  find  8  such  that  if  &  >  iV  and  Afc  <  8,  then  we  will  find 
an  acceptable  step.  In  view  of  the  algorithm  for  updating  Aa-  in  §2.4,  we  are  assured 
that  for  all  k  >  N ,  A^  >  08.  We  may  then  take  A*  =  min{  Aq,  •  ■  ■ ,  Ayv ,  08}.  D 

The  next  theorem  combines  the  strict  algebraic  structure  of  the  iterates  with  the 
simple  decrease  condition  of  the  generalized  pattern  search  algorithm  for  bound  con 
strained  problems,  along  with  the  algorithm  for  updating  Afc,  to  give  us  a  useful  fact 
about  the  limiting  behavior  of  Afc. 

Theorem  4.5.  Assume  that  Lq{xo)  is  compact.  Then  liminffc^+00  Ak  —  0. 

Proof  The  proof  is  like  that  of  Theorem  3.3  in  [14].  Suppose  0  <  Alb  <  ^k  for 
all  k.  Using  the  algorithm  for  updating  Afc  found  in  §2.4,  it  is  possible  to  write  Afc  as 
Afc  =  r’''=Ao,  where  rfc  €  Z. 

The  hypothesis  that  Alb  <  ^k  for  all  k  means  that  the  sequence  {r’’''}  is  bounded 
away  from  zero.  Meanwhile,  we  also  know  that  the  sequence  {Ak}  is  bounded  above 
because  all  the  iterates  Xk  must  lie  inside  the  set  Laixo)  =  {x  e  :  f{x)  <  f{xo)}  and 
the  latter  set  is  compact;  Lemma  2.1  then  guarantees  an  upper  bound  Aub  for  {Afc}. 
This,  in  turn,  means  that  the  sequence  {r'’*}  is  bounded  above.  Consequently,  the 
sequence  {t’’*}  is  a  finite  set.  Equivalently,  the  sequence  {rfc}  is  bounded  above  and 
below. 

Next  we  recall  the  exact  identity  of  the  quantities  riB  and  ruB  in  Theorem  2.2;  the 
details  are  found  in  the  proof  of  Theorem  3.3  in  [14].  In  the  context  of  Theorem  2.2, 


TLB 


min  {rfc} 
o<k<N  ^ 


ruB  = 


max  {rfc}. 
0<k<N  ^ 


If,  in  the  matter  at  hand,  we  let 

(13)  riB  =  min  {rfc}  ry/s  =  max  {rfc}, 

V  ''  0<fc<+oo  0<fc<+oo 
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then  (5)  holds  for  the  bounds  given  in  (13),  and  we  see  that  for  all  k,  Xk  lies  in  the 
translated  integer  lattice  G  generated  by  xo  and  the  columns  of  AoB. 

The  intersection  of  the  compact  set  Lq{xo)  with  the  lattice  G  is  finite.  Thus,  there 
must  exist  at  least  one  point  x,  in  the  lattice  for  which  Xk  =  x*  for  infinitely  many  k. 

We  now  appeal  to  the  simple  decrease  condition  in  Algorithm  1  (d),  which  guar¬ 
antees  that  an  iterate  cannot  be  revisited  infinitely  many  times  since  we  accept  a  new 
step  Sk  if  and  only  if  f{xk)  >  f{xk  +  5/t)  and  {xk  +  Sk)  €.  fi.  Thus  there  exists  an  N 
such  that  for  all  k  >  N,  Xk  =  x»,  which  implies  that  pk  =  0. 

We  now  appeal  to  the  algorithm  for  updating  A*,  (Algorithm  2  (a))  to  see  that 
Ak  0,  thus  leading  to  a  contradiction.  D 

4.1.  The  Proof  of  Theorem  3.2.  The  proof  is  like  that  of  Theorem  3.5  in  [14]. 

Suppose  that  lim  infA;_.+cc  ||  ||  0.  Then  Corollary  4.4  tells  us  that  there  exists 

A,  >  0  such  that  for  all  k,  Ak  >  A*.  But  this  contradicts  Theorem  4.5. 

4.2.  The  Proof  of  Theorem  3.3.  The  proof,  also  by  contradiction,  follows  that 
of  Theorem  3.7  in  [14].  Suppose  limsup,t^^oo  II  ||  ^  0.  Let  £  >  0  be  such  that 
there  exists  a  subsequence  ||  q{xmi)  ||  >  £•  Since 

liminf  II  ^(xfc)  II  =  0, 

given  any  0  <  7/  <  £,  there  exists  an  associated  subsequence  U  such  that 

II  II  >  ri  for  mi<k<  k,  ||  q{x^)  ||  <  rj. 

Since  Ak  0,  we  can  appeal  to  Proposition  4.3  to  obtain  for  mi  <  k  <  li,  i  sufficiently 
large. 


f{^k)-f{Xk+l)  >  II  9(2;^)  II  II  II  >  <T7/||sjt||, 


where  cr  >  0.  Then  the  telescoping  sum: 

(/(a^m.)  -  /(Xmi-n))  +  -  fiXmi+2))  +  •  •  •  +  ifixi,-l)  -  /(x^J)  >  ^  (77/  ||  Sk  || 

k=mi 

gives  US 

-  fM  >  YjUmi  II  Sk  II  >  c'  II  Xm,  -  Xl,  ||  . 

Since  /  is  bounded  below,  /(x„^,.)  -  /(x/,.)  ^  0  as  f  — >  +oo,  so  ||  Xm,  -  xi-  ||  ^  0 
as  i  —>■  +00.  Then,  because  q  is  uniformly  continuous,  ||  q{xm.i)  —  qix-u)  ||  <  7/,  for  i 
sufficiently  large.  However, 

(14)  11  q{Xmi)  II  A  II  q{Xmi)  ~  q{xii)  II  +  II  q{xi-)  II  <  27/. 

Since  (14)  must  hold  for  any  t/,  0  <  7/  <  £,  we  have  a  contradiction  (e.g.,  try  7/  =  |). 
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5.  Examples  of  Pattern  Search  Methods  for  Bound  Constrained  Min¬ 
imization,  A  section  of  [14]  is  devoted  to  showing  that  each  of  the  following  four 
algorithms: 

•  coordinate  search  with  fixed  step  lengths, 

•  evolutionary  operation  using  two-level  factorial  designs  ([1]  and  [3,  12]), 

•  the  original  pattern  search  method  of  Hooke  and  Jeeves  [9],  and 

•  the  multidirectional  search  algorithm  of  Dennis  and  Torczon  ([7]  and  [13]) 
are  pattern  search  methods  for  unconstrained  minimization.  In  this  section  we  will 
discuss  how  these  algorithms  may  be  extended  to  bound  constrained  problems.  We 
shall  see  that  coordinate  search  and  the  pattern  search  method  of  Hooke  and  Jeeves 
extend  without  modification  to  the  bound  constrained  case.  On  the  other  hand,  in  the 
case  of  multidirectional  search,  we  must  require  the  initial  basis  matrix  to  be  a  diagonal 
matrix  (in  the  unconstrained  case,  we  can  allow  any  nonsingular  basis  matrix);  in 
addition,  we  must  augment  the  columns  of  the  generating  matrix  to  ensure  a  sufficient 
set  of  search  directions.  In  the  case  of  evolutionary  operation,  we  also  must  augment 
the  columns  of  the  generating  matrix,  which  we  do  using  a  classical  variant  of  factorial 
designs  [2]. 

The  difference  between  pattern  search  methods  for  unconstrained  problems  and 
bound  constrained  problems  lies  in  the  two  additional  conditions  discussed  in  §2.5. 
First,  pattern  search  methods  for  bound  constrained  problems  must  start  with  a  feasible 
iterate  and  choose  feasible  trial  steps.  Second,  the  core  pattern  BMk  must  be  defined 
by  a  diagonal  matrix. 

We  assume  that  we  begin  with  a  feasible  iterate;  by  design  pattern  search  methods 
for  bound  constrained  problems  thereafter  accept  only  feasible  iterates.  Thus,  the  only 
thing  we  will  really  need  to  check  is  that  the  core  pattern  BMk  is  defined  by  a  diagonal 
matrix. 

It  is  this  latter  condition  that  causes  us  to  restrict  the  admissible  choice  of  the 
basis  matrix  in  multidirectional  search  and  then  augment  the  columns  of  the  generat¬ 
ing  matrix.  Moreover,  G.E.P.  Box’s  method  of  evolutionary  operation  using  two-level 
factorial  designs  does  not  satisfy  this  diagonality  condition;  we  will  present  a  simple 
counterexample  that  show  how  evolutionary  operation  can  fail  as  a  consequence  in  the 
bound  constrained  case  and  propose  a  straightforward  remedy  for  the  problem. 

5.1.  Coordinate  Search  and  the  Pattern  Search  Method  of  Hooke  and 
Jeeves.  Coordinate  search  and  the  pattern  search  method  of  Hooke  and  Jeeves  extend 
to  bound  constrained  problems  without  change.  In  both  cases  the  basis  matrix  B  is 
typically  chosen  to  be  a  diagonal  matrix:  either  the  identity  or  a  matrix  whose  entries 
reflect  the  relative  scaling  of  the  variables.  Furthermore,  the  first  3”  columns  of  Ck, 
which  are  fixed  for  all  iterations  k  of  both  algorithms,  are  composed  of  all  possible 
combinations  of  {—1,0,1}.  In  [14]  these  columns  are  organized  so  that  the  first  2n 
consist  of  the  identity  matrix  /  and  its  negative  —I.  In  terms  of  our  formalism,  then, 
Mk  =  I  for  all  iterations  k.  It  follows  that  BMk  is  a  diagonal  matrix,  as  required. 
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r9{xo) 


f{x)  <  f{xo) 


Fig.  1.  An  illustration  of  what  can  go  wrong  with  factorial  design  in  the  bound  constrained  case. 


5.2.  Evolutionary  Operation  Using  Factorial  Design.  For  the  evolutionary 
operation  algorithm  using  factorial  designs,  the  basis  matrix  B  is  usually  selected  to  be 
the  identity  or  a  diagonal  matrix  chosen  so  that  the  entries  along  the  diagonal  represent 
the  relative  scaling  among  the  variables.  However,  this  convention  is  not  sufficient  to 
ensure  that  BMk  is  a  diagonal  matrix. 

The  problem  lies  with  the  generating  matrix  C  =  [M  —M  L].  (The  generating 
matrix  C  is  fixed  across  all  iterations  of  evolutionary  operation.)  The  generating  matrix 
contains  in  its  columns  all  possible  combinations  of  {—1,1}  to  which  is  appended  a 
column  of  zeroes.  Clearly,  no  subset  of  n  columns  of  C  can  be  chosen  to  form  a  diagonal 
matrix  M. 

The  need  for  such  a  requirement  in  the  bound  constrained  case  can  be  seen  from 
the  following  example.  Consider  the  pattern  search  method  of  evolutionary  operation 
using  two-level  factorial  designs  applied  to  the  problem 

minimize  f{x)  =  — (.Ti  +  22:2) 
subject  to  0  <  Xi  <  1 
X2  <  0. 


Suppose  that  we  take  as  our  initial  iterate  the  point  (0,0);  the  pattern  for  factorial 
design  would  consist  of  the  points  indicated  by  the  open  circles  in  Figure  1  (we  show 
a  pattern  with  A^:  =  1/4).  We  see  that  for  any  choice  of  <  1  there  only  ever  will 
be  one  feasible  step  produced  by  the  pattern:  the  step  given  by  the  point  in  the  lower 
right-hand  corner  of  the  pattern— in  the  figure,  the  point  (1/4, -1/4) — and  that  this 
step  will  only  produce  increase  in  /.  So  evolutionary  operation  will  never  move  from 
its  starting  point. 

One  remedy  can  be  found  in  related  work  by  G.E.P.  Box  and  K.B.  Wilson  [2].  This 
would  be  to  use  a  composite  design  instead  of  the  two-level  factorial  design  usually 
proposed.  An  example  of  such  a  design — one  that  satisfies  the  requirements  of  the 

13 


bound  constrained  global  convergence  theory — chooses  M  to  be  the  diagonal  matrix 
with  entries  of  2  along  the  diagonal.  With  the  additional  2n  columns  that  would  then 
augment  the  original  pattern,  the  algorithm  would  not  fail  for  the  example  given  above, 
as  shown  in  Fig.  2.  We  now  have  a  feasible  step  along  the  active  constraint  X2  <0  that 
will  produce  descent  for  any  choice  of  Ai,  <  1  /2. 

As  noted  in  our  discussion  of  Proposition  4.2,  by  requiring  BMk  to  be  a  diagonal 
matrix,  we  are  guaranteed  that  the  directions  in  the  core  pattern  are  oriented  normal  and 
tangent  to  the  faces  of  the  feasible  region.  As  our  example  illustrates,  this  requirement 
is  essential. 

5.3.  Multidirectional  Search.  The  reader  should  be  forewarned  that  our  de¬ 
scription  and  discussion  of  multidirectional  search  takes  a  point  of  view  that  is  osten¬ 
sibly  at  odds  with  the  formalism  of  §2.1.  The  generating  matrix  T  is  viewed  as  fixed; 
typically  F  =  [M  —M]  =  [I  —I].  The  basis  matrix,  on  the  other  hand,  is  viewed  as 
varying  from  iteration  to  iteration  so  that  Bk  corresponds  to  the  edges  in  the  current 
simplex  that  are  adjacent  to  the  current  iterate  Xk-  This  is  the  reverse  of  the  discussion 
in  §2.1,  where  B  is  fixed  and  Tk  varies.  However,  the  former  view  of  multidirectional 
search  is  not  incompatible  with  the  formalism  of  pattern  search  methods,  as  noted  in 
[14],  and  as  we  shall  have  reason  to  discuss  here. 

The  extension  of  multidirectional  search  to  problems  with  bound  constraints  re¬ 
quires  us  to  restrict  the  choice  of  a  starting  simplex  and  to  augment  the  columns  of  the 
generating  matrix. 

The  first  restriction  is  minor  and  is  usually  satisfied  by  the  customary  choices  made 
in  practice.  In  multidirectional  search,  the  columns  of  Bo  are  formed  from  the  edges  of 
an  initial  simplex  adjacent  to  the  initial  iterate  xq.  In  the  case  of  bound  constraints, 
we  restrict  the  starting  simplex  to  be  a  right-angled  simplex,  i.e.,  the  n  -1- 1  vertices  of 
the  simplex  are  xq  and  the  points  Xq  +  where  G  R  and  i  —  I,  -  •  ■  ,n.  Because  of 
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this  choice,  Bq  =  diag(Q;,).  Since  M  =  I,  the  product  BqM  is  a  diagonal  matrix. 

However,  even  if  the  initial  simplex  is  restricted  to  be  a  right-angled  simplex  so 
that  BqM  is  diagonal,  there  is  no  guarantee  that  in  subsequent  iterations  BkM  will  be 
diagonal.  To  understand  why  this  is  so,  and  how  this  may  be  corrected  by  augmenting 
the  columns  of  the  generating  matrix,  we  need  to  discuss  how  multidirectional  search 
fits  within  the  formalism  of  pattern  search  methods.  These  details  are  absent  from  [14], 
so  we  present  them  here. 

At  iteration  k,  the  basis  matrix  is 


where  vl,  i  =  0,  •  •  •  ,n  are  the  vertices  of  the  simplex  associated  with  multidirectional 
search  at  this  iteration.  Define 


i  =  0 

;) 


,  n. 


Now  consider  what  happens  in  the  next  iteration.  If  the  iteration  is  unsuccessful,  then 
^°+i  =  ‘^k  fhe  new  basis  for  the  pattern,  which  is  determined  by  the  edges  of  the 
simplex  emanating  from  is 


Bk+i  =  Bk  =  BkTo- 

If,  on  the  other  hand,  the  iteration  is  successful,  then  —  (vl  —  n°)  for  some 

j  =  1, . . . ,  n,  and  the  new  basis  will  be  the  set  of  vectors 

_  f  bl  if  i  =  j 
\  —bl  +  b{  otherwise. 

In  this  case. 


Bk+i  =  BkTj. 

Thus,  in  general, 

(15)  = 

and  so 

(16)  =  Bt.2Ti,_,T„  =  ...  =  B„nr,,. 

t  =  l 

Our  next  goal  is  to  simplify  this  relation  further. 

A  short  calculation  shows  that  for  i,t  =  1, . . . ,  n, 

TiTi  =  I  -  e(ej  -  ^  -  e^-ef  +  S}eieJ  +  ^  +  aej , 
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where  6\  is  the  Kronecker  delta.  If  i  =  ^,  this  reduces  to 
(17)  TiTi  =  I 

and  \i  i  ^  I, 


(18) 

If  i  ^  C, 

(19) 

and 

(20) 


TiT^  =  I  —  e^ej  —  ^  —  Cicf  +  e^-ej  —  —T^  —  e^-ef  + 


m=l 


Teei  =  -  ll  -  eeej  -  ^  )  e*-  =  -  e,- 


m=l 


(r^-  (— ^Ti?))  —  I  —I  +  e£ej  +  ^  +  e^-ef  —  e^-ej  j  e^-  —  e^. 


771  =  1 


From  (17),  (18),  and  (19)  we  obtain 

TeTiTi  =  Ti  [—Ti  —  e,ef  +  e,ej) 

=  -I  -  TeeieJ  +  TetitJ 
=  -I  -  {e£  -  ei)ej  +  {a  -  ei)ej 
(21)  =  -  (7  -  e,ef  -  e/tej  +  e^ef  +  CjeJ)  . 

This  latter  matrix  we  recognize  as  the  negative  of  the  elementary  permutation  matrix 
E{i,  £)  that  swaps  the  f*'*  and  £*'^  columns  when  acting  on  matrices  from  the  right.  Using 
(17)  and  (21),  we  obtain  the  rule 


TiT,  = 


/  \ii  =  £ 
—T(E{i,  £)  otherwise. 


We  can  then  use  this  formula  to  reduce  (16)  to 

Bk  =  ±I9oT4nfc, 


for  some  T4  and  permutation  matrix  Ilfc. 

This  relationship  reveals  several  things.  The  first  is  that  it  reconciles  the  usual 
description  of  multidirectional  search  with  the  formal  abstract  definition  of  a  pattern 
search  method;  the  pattern  matrix  is  given  by 


(22)  BkC  =  ±B^T,Jlk[I  -I  0]  =  B^[Ti,  -  T,,  0]n,  =  BCk- 

That  is,  we  may  interpret  multidirectional  search  in  terms  of  a  fixed  basis  B  and  a 
changing  generating  matrix  Ck- 
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We  can  also  see  that  while  BFo  will  be  diagonal,  this  diagonality  may  be  lost  in 
subsequent  iterations.  However,  the  form  of  the  generic  pattern  from  the  unconstrained 
algorithm  suggests  one  way  to  circumvent  this  problem  in  the  bound  constrained  case. 
This  remedy  will,  moreover,  preserve  the  geometric  interpretation  of  the  pattern  in 
multidirectional  search  in  terms  of  a  simplex. 

First,  if  we  ignore  the  permutation  in  (22),  which  only  affects  column  ordering,  the 
pattern  at  iteration  k  in  the  unconstrained  case  is  given  by 

B,C  =  BCk  =  Bo[T^,  -r,,  0]. 

Suppose  we  augment  the  columns  of  C  to  include  all  the  Tf. 

C  =  [-To  -T, - Tn  0]. 

At  any  iteration  k,  up  to  a  column  permutation,  the  basis  matrix  is  the  matrix  Bk  = 
jk  G  {0,  •  •  • ,  n}.  When  we  then  form  the  pattern  Pk  =  AkBkC,  we  have 

Pk  =  A,BkC  =  AkB[±T,Jo±T,Jr---±TjJ„]  =  AkBCk- 

Now  note  that  (20)  means  that  for  jk  ^  /  the  column  of  -Tj^Te  is  the  basis 
vector.  Consequently,  we  are  guaranteed  that  by  a  permutation  of  the  columns  of  Ck, 

Ck  =  [I  -I  Lk]  =  [T  Lkl 

where  Lk  changes  at  each  iteration,  but  F  does  not.  Since  we  require  the  initial  simplex 
to  be  a  right-angled  simplex,  we  may  then  be  assured  that  BF  =  [diag(Qi)  -diag(cii)], 
as  required. 

Moreover,  this  augmentation  of  C  and  the  search  through  its  columns  can  be  imple¬ 
mented  in  a  way  that  preserves  the  relationship  of  the  pattern  to  the  moving  simplex 
that  characterizes  multidirectional  search.  This  is  possible  because  the  matrices  Z), 

*  =  0,  ■  •  • ,  n  capture  how  the  basis  changes  in  association  with  a  change  of  simplex. 
This  is  the  gist  of  (15).  The  implications  for  any  implementation  of  this  modification 
to  multidirectional  search  to  handle  bound  constraints  will  appear  elsewhere. 

6.  Conclusion.  We  have  presented  a  reasonable  extension  of  pattern  search  meth¬ 
ods  for  unconstrained  minimization  to  bound  constrained  problems.  The  extension  is 
supported  by  a  global  convergence  theory  as  strong  as  that  for  the  unconstrained  case. 
The  generalization  makes  few  additional  requirements  and  as  we  have  seen  in  §5,  the 
classical  pattern  search  methods  for  unconstrained  minimization  or  straightforward  vari¬ 
ants  thereof  carry  over  to  the  bound  constrained  case. 

The  extension  to  bound  constrained  problems  also  points  the  way  to  handling 
general  linear  inequalities.  General  linear  inequalities  will  require  a  sufficiently  rich  set 
of  directions  in  the  pattern  so  that  at  any  face  of  the  feasible  region,  one  will  have 
directions  both  normal  and  tangent  to  the  constraints.  This  we  will  pursue  elsewhere. 

One  issue  we  have  not  discussed  is  that  of  identifying  active  constraints,  as  in 
[4,  5].  One  would  wish  to  show  that  if  the  sequence  {xk]  converges  to  a  nondegenerate 
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stationary  point  x,,  then  in  a  finite  number  of  iterations  the  iterates  Xk  land  on  the 
constraints  active  at  x,  and  remain  thereafter  on  those  constraints. 

There  are  three  difficulties  in  proving  such  a  result  for  pattern  search  methods  for 
bound  constrained  minimization.  The  first  is  relatively  minor.  If  the  iterates  Xk  are 
to  identify  the  active  constraints  for  a  stationary  point  on  the  boundary  of  the  feasible 
region,  we  must  ensure  that  the  lattice  manifest  in  Theorem  2.2  actually  allows  iterates 
to  land  on  the  boundary.  This  requires  additional  but  straightforward  conditions  on 
such  quantities  as  xo,t,  Aq,  and  the  pattern  matrices  Pk-  A  related  but  more  subtle 
difficulty  is  that  the  relative  sizes  of  the  steps  in  the  core  pattern  and  the  remaining 
points  in  the  pattern  must  obey  certain  relations  in  order  to  ensure  that  the  algorithm 
does  not  take  a  purely  interior  approach  to  a  point  on  the  boundary.  This  rules  out,  for 
instance,  certain  of  the  composite  designs  suggested  by  G.E.P.  Box  and  K.B.  Wilson 
[2]. 

The  most  serious  obstacle,  which  remains  to  be  overcome,  is  showing  that  ultimately 
the  iterates  will  land  on  the  active  constraints  and  remain  there.  For  algorithms  such  as 
those  considered  in  [4,  5],  this  is  not  a  problem  because  the  explicit  use  of  the  gradient 
impels  the  iterates  to  do  this  in  the  neighborhood  of  a  nondegenerate  stationary  point. 
However,  pattern  search  methods  do  not  have  this  information.  On  the  other  hand,  the 
kinship  of  pattern  search  methods  and  gradient  projection  methods  makes  us  hopeful 
that  ultimately  we  will  be  able  to  prove  that  pattern  search  methods  also  identify  the 
active  constraints  in  a  finite  number  of  iterations. 
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